Field of the Invention
The present invention relates to an image processing apparatus, an image capturing apparatus, a control method of the image processing apparatus, and a storage medium.
Description of the Related Art
Conventionally, omnidirectional cameras capable of shooting 360 degree (omnidirectional) images are known. For example, Japanese Patent Laid-Open No. 6-501585 discloses a technique for converting a hemispherical field-of-view image (omnidirectional image) into a normal perspective image that is not distorted at an arbitrary orientation (direction), rotation and magnification within the field of view. According to Japanese Patent Laid-Open No. 6-501585, a subject image at an arbitrary position in a visual field space within a hemispherical field of view can be provided as a moving image without moving a mechanical panning/tilting mechanism. In addition, among the above parameters (orientation (direction), rotation, and magnification), magnification corresponds to a focal distance at the time of image conversion. In other words, decreasing the magnification enables the actual focal distance to be shortened, making it possible to cover a wide angle of view. In general, when attempting to cover a wide angle of view, a phenomenon occurs in which the enlargement amount of the subject image varies between an area positioned at the periphery of the hemispherical field of view and an area positioned at the center.
In addition, Japanese Patent Laid-Open No. 6-501585 discloses a fixed-type camera apparatus to which the above technique is applied, but the above technique can also be applied to the case where a camera apparatus is installed in a vehicle or worn by a person as a wearable apparatus. In such cases, there is a possibility that shake of the camera apparatus caused by the vehicle or the person will occur.
There are conventionally known techniques regarding moving image shake correction. For example, Japanese Patent Laid-Open No. 2007-221631 discloses a technique for performing shake correction by detecting motion vectors in captured images in units of screens from captured image data, and shifting a position for reading out captured image data stored in an image memory based on these motion vectors. In addition, Japanese Patent Laid-Open No. 2009-258868 discloses a technique for calculating a global motion indicating the movement of the entire image between two screens using a large number of local motion vectors. Specifically, a motion vector (local motion vector) is detected for each of a large number of target blocks set in the screen, and the global motion is calculated as an affine transformation matrix using the motion vectors.
In a hemispherical field-of-view image, a large amount of distortion occurs particularly in the peripheral area, and therefore shapes vary in accordance with position even with the same subject. Therefore, in the case of detecting local motion vectors for the hemispherical field-of-view image, there is a possibility that the accuracy of block matching processing (processing for detecting highly correlated blocks between two screens) will deteriorate as the subject moves, and that the accuracy of detecting the local motion vectors will deteriorate. In view of this, it is conceivable to correct the distortion by converting the target blocks of the hemispherical field-of-view image in accordance with the technique of Japanese Patent Laid-Open No. 6-501585.
However, in the case of performing distortion correction, the enlargement amount of the subject image varies in accordance with the position in the hemispherical field-of-view image. Also, the movement direction of the subject in a distortion corrected image does not necessarily match the movement direction of the same subject in the hemispherical field-of-view image (before the distortion correction). Therefore, even if the actual movement amount and the movement direction of the subject are the same, there is a possibility that local motion vectors having a size and orientation that vary in accordance with the position of the target block are detected. That is, the local motion vectors detected in the hemispherical field-of-view image in accordance with the techniques of Japanese Patent Laid-Open No. 6-501585 and Japanese Patent Laid-Open No. 2009-258868 do not appropriately reflect the movement of the subject in the hemispherical field-of-view image (before distortion correction) in some cases. This problem can similarly occur to varying degrees in a captured image having an angle of view narrower than the angle of view of the hemispherical field-of-view image.